{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "# Plot parameters\n",
    "sns.set()\n",
    "%pylab inline\n",
    "pylab.rcParams['figure.figsize'] = (4, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Avoid inaccurate floating values (for inverse matrices in dot product for instance)\n",
    "# See https://stackoverflow.com/questions/24537791/numpy-matrix-inversion-rounding-errors\n",
    "np.set_printoptions(suppress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".pquote {\n",
       "  text-align: left;\n",
       "  margin: 40px 0 40px auto;\n",
       "  width: 70%;\n",
       "  font-size: 1.5em;\n",
       "  font-style: italic;\n",
       "  display: block;\n",
       "  line-height: 1.3em;\n",
       "  color: #5a75a7;\n",
       "  font-weight: 600;\n",
       "  border-left: 5px solid rgba(90, 117, 167, .1);\n",
       "  padding-left: 6px;\n",
       "}\n",
       ".notes {\n",
       "  font-style: italic;\n",
       "  display: block;\n",
       "  margin: 40px 10%;\n",
       "}\n",
       "img + em {\n",
       "  text-align: center;\n",
       "  display: block;\n",
       "  color: gray;\n",
       "  font-size: 0.9em;\n",
       "  font-weight: 600;\n",
       "}\n",
       "</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<style>\n",
    ".pquote {\n",
    "  text-align: left;\n",
    "  margin: 40px 0 40px auto;\n",
    "  width: 70%;\n",
    "  font-size: 1.5em;\n",
    "  font-style: italic;\n",
    "  display: block;\n",
    "  line-height: 1.3em;\n",
    "  color: #5a75a7;\n",
    "  font-weight: 600;\n",
    "  border-left: 5px solid rgba(90, 117, 167, .1);\n",
    "  padding-left: 6px;\n",
    "}\n",
    ".notes {\n",
    "  font-style: italic;\n",
    "  display: block;\n",
    "  margin: 40px 10%;\n",
    "}\n",
    "img + em {\n",
    "  text-align: center;\n",
    "  display: block;\n",
    "  color: gray;\n",
    "  font-size: 0.9em;\n",
    "  font-weight: 600;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\newcommand\\bs[1]{\\boldsymbol{#1}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span class='notes'>\n",
    "    This content is part of a series following the chapter 2 on linear algebra from the [Deep Learning Book](http://www.deeplearningbook.org/) by Goodfellow, I., Bengio, Y., and Courville, A. (2016). It aims to provide intuitions/drawings/python code on mathematical theories and is constructed as my understanding of these concepts. You can check the syllabus in the [introduction post](https://hadrienj.github.io/posts/Deep-Learning-Book-Series-Introduction/).\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "This chapter is light but contains some important definitions. The identity matrix or the inverse of a matrix are concepts that will be very useful in the next chapters. We will see at the end of this chapter that we can solve systems of linear equations by using the inverse matrix. So hang on!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.3 Identity and Inverse Matrices\n",
    "\n",
    "\n",
    "# Identity matrices\n",
    "\n",
    "The identity matrix $\\bs{I}_n$ is a special matrix of shape ($n \\times n$) that is filled with $0$ except the diagonal that is filled with 1.\n",
    "\n",
    "<img src=\"images/identity-matrix.png\" width=\"150\" alt=\"Example of an identity matrix\" title=\"Identity matrix\">\n",
    "<em>A 3 by 3 identity matrix</em>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An identity matrix can be created with the Numpy function `eye()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  0.,  0.],\n",
       "       [ 0.,  1.,  0.],\n",
       "       [ 0.,  0.,  1.]])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.eye(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When 'apply' the identity matrix to a vector the result is this same vector:\n",
    "\n",
    "$$\\bs{I}_n\\bs{x} = \\bs{x}$$\n",
    "\n",
    "### Example 1.\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    "    1 & 0 & 0 \\\\\\\\\n",
    "    0 & 1 & 0 \\\\\\\\\n",
    "    0 & 0 & 1\n",
    "\\end{bmatrix}\n",
    "\\times\n",
    "\\begin{bmatrix}\n",
    "    x_{1} \\\\\\\\\n",
    "    x_{2} \\\\\\\\\n",
    "    x_{3}\n",
    "\\end{bmatrix}=\n",
    "\\begin{bmatrix}\n",
    "    1 \\times x_1 + 0 \\times x_2 + 0\\times x_3 \\\\\\\\\n",
    "    0 \\times x_1 + 1 \\times x_2 + 0\\times x_3 \\\\\\\\\n",
    "    0 \\times x_1 + 0 \\times x_2 + 1\\times x_3\n",
    "\\end{bmatrix}=\n",
    "\\begin{bmatrix}\n",
    "    x_{1} \\\\\\\\\n",
    "    x_{2} \\\\\\\\\n",
    "    x_{3}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2],\n",
       "       [6],\n",
       "       [3]])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.array([[2], [6], [3]])\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2.],\n",
       "       [ 6.],\n",
       "       [ 3.]])"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xid = np.eye(x.shape[0]).dot(x)\n",
    "xid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intuition\n",
    "\n",
    "You can think of a matrix as a way to transform objects in a $n$-dimensional space. It applies a linear transformation of the space. We can say that we *apply* a matrix to an element: this means that we do the dot product between this matrix and the element (more details about the dot product in [2.2](https://hadrienj.github.io/posts/Deep-Learning-Book-Series-2.2-Multiplying-Matrices-and-Vectors/)). We will see this notion thoroughly in the next chapters but the identity matrix is a good first example. It is a particular example because the space doesn't change when we *apply* the identity matrix to it.\n",
    "\n",
    "<span class='pquote'>\n",
    "    The space doesn't change when we *apply* the identity matrix to it\n",
    "</span>\n",
    "\n",
    "We saw that $\\bs{x}$ was not altered after being multiplied by $\\bs{I}$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inverse Matrices\n",
    "\n",
    "The matrix inverse of $\\bs{A}$ is denoted $\\bs{A}^{-1}$. It is the matrix that results in the identity matrix when it is multiplied by $\\bs{A}$:\n",
    "\n",
    "$$\\bs{A}^{-1}\\bs{A}=\\bs{I}_n$$\n",
    "\n",
    "This means that if we apply a linear transformation to the space with $\\bs{A}$, it is possible to go back with $\\bs{A}^{-1}$. It provides a way to cancel the transformation.\n",
    "\n",
    "### Example 2.\n",
    "\n",
    "$$\n",
    "\\bs{A}=\\begin{bmatrix}\n",
    "    3 & 0 & 2 \\\\\\\\\n",
    "    2 & 0 & -2 \\\\\\\\\n",
    "    0 & 1 & 1\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "For this example, we will use the Numpy function `linalg.inv()` to calculate the inverse of $\\bs{A}$. Let's start by creating $\\bs{A}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 3,  0,  2],\n",
       "       [ 2,  0, -2],\n",
       "       [ 0,  1,  1]])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = np.array([[3, 0, 2], [2, 0, -2], [0, 1, 1]])\n",
    "A"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we calculate its inverse:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.2,  0.2,  0. ],\n",
       "       [-0.2,  0.3,  1. ],\n",
       "       [ 0.2, -0.3, -0. ]])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A_inv = np.linalg.inv(A)\n",
    "A_inv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check that $\\bs{A_{inv}}$ is well the inverse of $\\bs{A}$ with Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.,  0., -0.],\n",
       "       [ 0.,  1., -0.],\n",
       "       [ 0.,  0.,  1.]])"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A_bis = A_inv.dot(A)\n",
    "A_bis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will see that inverse of matrices can be very usefull, for instance to solve a set of linear equations. We must note however that non square matrices (matrices with more columns than rows or more rows than columns) don't have inverse."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sovling a system of linear equations\n",
    "\n",
    "An introduction on system of linear equations can be found in [2.2](https://hadrienj.github.io/posts/Deep-Learning-Book-Series-2.2-Multiplying-Matrices-and-Vectors/).\n",
    "\n",
    "The inverse matrix can be used to solve the equation $\\bs{Ax}=\\bs{b}$ by adding it to each term:\n",
    "\n",
    "$$\\bs{A}^{-1}\\bs{Ax}=\\bs{A}^{-1}\\bs{b}$$\n",
    "\n",
    "Since we know by definition that $\\bs{A}^{-1}\\bs{A}=\\bs{I}$, we have:\n",
    "\n",
    "$$\\bs{I}_n\\bs{x}=\\bs{A}^{-1}\\bs{b}$$\n",
    "\n",
    "We saw that a vector is not changed when multiplied by the identity matrix. So we can write:\n",
    "\n",
    "$$\\bs{x}=\\bs{A}^{-1}\\bs{b}$$\n",
    "\n",
    "This is great! We can solve a set of linear equation just by computing the inverse of $\\bs{A}$ and apply this matrix to the vector of results $\\bs{b}$!\n",
    "\n",
    "Let's try that!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3.\n",
    "\n",
    "We will take a simple solvable example:\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "y = 2x \\\\\\\\\n",
    "y = -x +3\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "We will use the notation that we saw in [2.2](https://hadrienj.github.io/posts/Deep-Learning-Book-Series-2.2-Multiplying-Matrices-and-Vectors/):\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "A_{1,1}x_1 + A_{1,2}x_2 = b_1 \\\\\\\\\n",
    "A_{2,1}x_1 + A_{2,2}x_2= b_2\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "Here, $x_1$ corresponds to $x$ and $x_2$ corresponds to $y$. So we have:\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "2x_1 - x_2 = 0 \\\\\\\\\n",
    "x_1 + x_2= 3\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "Our matrix $\\bs{A}$ of weights is:\n",
    "\n",
    "$$\n",
    "\\bs{A}=\n",
    "\\begin{bmatrix}\n",
    "    2 & -1 \\\\\\\\\n",
    "    1 & 1\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "And the vector $\\bs{b}$ containing the solutions of individual equations is:\n",
    "\n",
    "$$\n",
    "\\bs{b}=\n",
    "\\begin{bmatrix}\n",
    "    0 \\\\\\\\\n",
    "    3\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Under the matrix form, our systems becomes:\n",
    "\n",
    "$$\n",
    "\\begin{bmatrix}\n",
    "    2 & -1 \\\\\\\\\n",
    "    1 & 1\n",
    "\\end{bmatrix}\n",
    "\\begin{bmatrix}\n",
    "    x_1 \\\\\\\\\n",
    "    x_2\n",
    "\\end{bmatrix}=\n",
    "\\begin{bmatrix}\n",
    "    0 \\\\\\\\\n",
    "    3\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Let's find the inverse of $\\bs{A}$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 2, -1],\n",
       "       [ 1,  1]])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A = np.array([[2, -1], [1, 1]])\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.33333333,  0.33333333],\n",
       "       [-0.33333333,  0.66666667]])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "A_inv = np.linalg.inv(A)\n",
    "A_inv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have also:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "b = np.array([[0], [3]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we saw that\n",
    "\n",
    "$$\\bs{x}=\\bs{A}^{-1}\\bs{b}$$\n",
    "\n",
    "We have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.],\n",
       "       [ 2.]])"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = A_inv.dot(b)\n",
    "x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is our solution! \n",
    "\n",
    "$$\n",
    "\\bs{x}=\n",
    "\\begin{bmatrix}\n",
    "    1 \\\\\\\\\n",
    "    2\n",
    "\\end{bmatrix}\n",
    "$$\n",
    "\n",
    "This means that the point of coordinates (1, 2) is the solution and is at the intersection of the lines representing the equations. Let's plot them to check this solution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQkAAAECCAYAAADtryKnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFSJJREFUeJzt3X+M1PWdx/Hncqse9ZYfV7fkbDH0Svct6YKRerSnwAm1\n19ZK2qaXXEzUlhSoJ7mcGpvA9ej1j2shkTPRFJoKFn/Us2pTr1dp1KSgwF7O9o8GQelbLD3bpEZX\nBUHRdHHn/piZ88u4O/Odme+Pz/c7r0eyCTvf3Zm3X+HJm/mx01epVBARmcyUvAcQkbApEiLSlCIh\nIk0pEiLSlCIhIk0pEiLSVH+rLzCzKcA2wIBx4Fp3fyZyfAWwARgDdrj79pRmFZEcxNkkVgAVd19M\nNQbfrh8ws37gFuAy4FJgjZm9L4U5RSQnLSPh7j8B1tQ+nQMcjRyeBxx29+PuPgbsA5YkPaSI5Kfl\nPzcA3H3czO4EPg/8XeTQNOC1yOcngOmJTSciuYt9x6W7fxkYArab2dTaxcephqJuADiW2HQikrs4\nd1xeBXzA3TcBbwFv1z4ADgFzzWwGcBJYCtzc7Pr+5dvfqPzv0DHWLVnL8CzranjJRqVSYf3WEZ4+\n8gr//k9LGTpvZt4jSWf6OvmmOP/c+DGww8yeqH399cAXzexsd99uZjcCj9UG2O7uLzS7smlnDTA+\n/iob92zhugtWMjRzbidzJ2pwcIDR0RN5j3GakGY69PxRnj7yCgs+9F6GzpsZzFx1IZ2rulBn6kTL\nSLj7SeDvmxzfCeyMe4Nn/skZrJ5/DdsO3M3W/TuCCYVMrFKp8JN9vwXgc4s/mPM0kodcnkw1fM48\nVs+/hkplnK37d/Ds0efyGENi+PXvjvHs74+x4EPv5YN/Ma31N0jp5PaMS4UifNoiBHJ+WrZCETZt\nEQIBvHZDoQiTtgipyz0SoFCESFuE1AURCVAoQqItQqKCiQQoFKHQFiFRQUUCFIq8aYuQRsFFAhSK\nPGmLkEZBRgIUijxoi5CJBBsJUCiypi1CJhJ0JEChyIq2CJlM8JEAhSIL2iJkMoWIBCgUadIWIc0U\nJhKgUKRFW4Q0U6hIgEKRNG0R0krhIgEKRZK0RUgrhYwEKBRJ0BYhcRQ2EqBQdEtbhMRR6EiAQtEp\nbRESV+EjAQpFJ7RFSFyliAQoFO3QFiHtKE0kQKGIS1uEtKNUkQCFohVtEdKu0kUCFIpmtEVIu0oZ\nCVAoJqItQjpR2kiAQtFIW4R0otSRAIWiTluEdKr0kQCFArRFSOeavqu4mfUD3wfmAGcC33L3n0aO\n3wB8BXipdtFX3f1wOqN2px6KXnw3c20R0o1Wm8RVwMvuvhS4HPhOw/GFwNXuvrz2EWQg6np1o9AW\nId1oFYkHgA21X/cBYw3HPwqsN7O9ZrYu6eHS0Guh0BYh3WoaCXc/6e5vmNkA8CDw9YYvuQ+4FlgG\nLDazy9MZM1mNoTj4ouc9Umq0RUi3Wt5xaWazgV3AXe5+f8PhW939VXc/BewELkxhxlREQ7Fp75ZS\nbhTaIiQJre64nAU8Cqx1990Nx6YBB83sfOBNYDlwR5wbHRwc6GzahC0bXMT06VPZPHI7331qB+uW\nrGV4luU91v/r9jw99dwoz/7+GBfNm8WiBe9PaKpw/v9Faab0NI0EsB6YAWwws28AFWAbcLa7bzez\n9cDjwFvAz939kTg3Ojp6ovOJEzb7jDncdMkaNu/7Hhv3bAnmUY/BwYGuzlOlUuGuh58B4DOLZid2\nzrudKw2aKZ5Oo9U0Eu5+PXB9k+P3Avd2dMsBWXju/NI9PKr7IiQpPfFkqjjK9KiH7ouQJCkSEWUJ\nhbYISZIi0aDoodAWIUlTJCZQ5FBoi5CkKRKTKGIotEVIGhSJJooWCm0RkgZFooWihEJbhKRFkYih\nCKHQFiFpUSRiCjkU2iIkTYpEG0INhbYISZMi0abQQqEtQtKmSHQgpFBoi5C0KRIdCiEU2iIkC4pE\nF/IOhbYIyYIi0aW8QqEtQrKiSCQgj1Boi5CsKBIJyTIU2iIkS4pEgrIKhbYIyZIikbC0Q6EtQrKm\nSKQgzVBoi5CsKRIpSSMU2iIkD4pEipIOhbYIyYMikbKkQqEtQvKiSGQgiVBoi5C8KBIZ6SYU2iIk\nT4pEhjoNhbYIyZMikbF2Q6EtQvKmSOSgnVBoi5C8KRI5iRMKbRESAkUiR61CoS1CQtA0EmbWb2Z3\nm9keM/sfM1vRcHyFmf3CzEbMbFW6o5bTZKHQFiGhaLVJXAW87O5LgcuB79QPmFk/cAtwGXApsMbM\n3pfSnKU2USgO/OZlbREShFaReADYUPt1HzAWOTYPOOzux919DNgHLEl+xN7QGIo7dj0BaIuQ/DWN\nhLufdPc3zGwAeBD4euTwNOC1yOcngOnJj9g76qF4e3ycP/zZbubamLYIyV1/qy8ws9nAj4HvuPv9\nkUPHqYaibgA4FudGBwcH2pkxE6HMdOk5f8VDe3/LCwNP8NLMPbw4Pp/hWZb3WKcJ5VxFaab0NI2E\nmc0CHgXWuvvuhsOHgLlmNgM4CSwFbo5zo6OjJzoYNT2DgwPBzHTo+aP81v8UG17OH85+nI17tnDd\nBSsZmjk379GAsM5VnWaKp9NotbpPYj0wA9hgZrvNbJeZXWlmq9z9FHAj8BgwAmx39xc6mkKA0x/R\nWPOJT+T+vh4i0GKTcPfrgeubHN8J7Ex6qF4VfV7E0HkzmTlavY9i24G72bp/R1AbhfQOPZkqEJM9\nLyLvNwASUSQC0ezZlQqF5EmRCECcZ1cqFJIXRSIAcV+joVBIHhSJnLX7Gg2FQrKmSOSsk1d6KhSS\nJUUiR9280lOhkKwoEjnq9udFKBSSBUUiJ0n9vAiFQtKmSOQkyZ86pVBImhSJHKTxU6cUCkmLIpGD\ntH52pUIhaVAkMpb2z65UKCRpikTGsvgJ2AqFJEmRyFCWPwFboZCkKBIZyvp9NBQKSYIikZG83kdD\noZBuKRIZyfPduBQK6YYikYEQ3o1LoZBOKRIZCOU9PRUK6YQikbIQtogohULapUikLJQtIkqhkHYo\nEikKbYuIUigkLkUiRSFuEVEKhcShSKQk5C0iSqGQVhSJlIS+RUQpFNKMIpGComwRUQqFTEaRSEGR\ntogohUImokgkrIhbRJRCIY1iRcLMPmZmuye4/AYzO2hmu2ofH05+xGIp6hYRpVBIVMtImNnXgG3A\nWRMcXghc7e7Lax+Hkx6wSIq+RUQpFFIXZ5N4DvjCJMc+Cqw3s71mti65sYqpDFtElEIhECMS7v4Q\ncGqSw/cB1wLLgMVmdnmCsxVKmbaIqMZQHHzR8x5JMtbf5fff6u7HAcxsJ3Ah8LNW3zQ4ONDlzSav\n25meem6UZ39/jIvmzWLRgvcHMVNSlg0uYvr0qWweuZ1Ne7ewbslahmdZ3mOdJpRzFRXiTJ1oJxJ9\n0U/MbBpw0MzOB94ElgN3xLmi0dETbdxs+gYHB7qaqVKpcNfDzwDwmUWzE/nv63ampM0+Yw6rh69m\n28F72LhnC9ddsJKhmXPzHgsI71xBuDN1op2HQCsAZnalma2qbRDrgceBJ4CD7v5IR1MUXNnui5jM\n8DnzuOmSNbqPosfE2iTc/Xng4tqv74tcfi9wbzqjFUNZ74uYzMJz57N6/jVsO3A3W/fvCGqjkHTo\nyVRd6pUtIkqPevQWRaILvbZFRCkUvUOR6EIvbhFRCkVvUCQ61MtbRJRCUX6KRId6fYuIUijKTZHo\ngLaId1MoykuR6IC2iIkpFOWkSLRJW0RzCkX5KBJt0hbRmkJRLopEG7RFxKdQlIci0QZtEe1RKMpB\nkYhJW0RnFIriUyRi0hbROYWi2BSJGLRFdE+hKC5FIgZtEclQKIpJkWhBW0SyFIriUSRa0BaRPIWi\nWBSJJrRFpEehKA5FogltEelSKIpBkZiEtohsKBThUyQmoS0iOwpF2BSJCWiLyJ5CES5FYgLaIvKh\nUIRJkWigLSJfCkV4FIkG2iLyp1CERZGI0BYRDoUiHIpEhLaIsCgUYVAkarRFhEmhyJ8iUaMtIlwK\nRb5iRcLMPmZmuye4fIWZ/cLMRsxsVfLjZUNbRPgUivy0jISZfQ3YBpzVcHk/cAtwGXApsMbM3pfC\njKk78JuXtUUUgEKRjzibxHPAFya4fB5w2N2Pu/sYsA9YkuRwWahUKvzHow5oiygChSJ7LSPh7g8B\npyY4NA14LfL5CWB6QnNl5te/O8bTR17RFlEgjaE4+KLnPVKp9XfxvcephqJuADgW5xsHBwe6uNnk\nVCoVfvbAfgC+dMVHgpmrLrR56kKYa9ngIqZPn8rmkdvZtHcL65asZXiW5T3WaUI4T0loJxJ9DZ8f\nAuaa2QzgJLAUuDnOFY2OnmjjZtNz6PmjPH3kFS6aN4uZU/uDmQuqv8FCmqcupLlmnzGH1cNXs+3g\nPWzcs4XrLljJ0My5eY8FhHWe6jqNVjsPgVYAzOxKM1vl7qeAG4HHgBFgu7u/0NEUOYg+onHl34b1\nN5DEN3zOPG66ZI3uo0hRrE3C3Z8HLq79+r7I5TuBnemMlq7o8yKGzpsZXPUlvoXnzmf1/GvYduBu\ntu7fEdRGUQY9+WQqPS+ifPSoR3p6MhJ6dmU5KRTp6LlIaIsoN4UieT0XCW0R5adQJKunIqEtonco\nFMnpqUhoi+gtCkUyeiYS2iJ6k0LRvZ6JhLaI3qVQdKcnIqEtQhSKzvVEJLRFCCgUnSp9JLRFSJRC\n0b7SR0JbhDRSKNpT6khoi5DJKBTxlToS2iKkGYUintJGQluExKFQtFbaSGiLkLgUiuZKGQltEdIu\nhWJypYyEtgjphEIxsdJFQluEdEOheLfSRUJbhHRLoThdqSKhLUKSolC8o1SR0BYhSVIoqkoTCW0R\nkgaFokSR0BYhaen1UJQiEtoiJG29HIpSREJbhGShV0NR+Ehoi5As9WIoCh8JbRGStV4LRaEjoS1C\n8tJLoWj5ruJm1gdsBS4A3gJWufuRyPFbqb7jeP1tuT/n7pm8Rbe2CMlTPRRlfzfzOJvE54Gz3P1i\nYD1wS8PxhcCn3H157SOTQGiLkBD0wkYRJxKLgUcA3P1J4KL6gdqW8WHgdjPbZ2YrU5lyAtoiJBRl\nD0WcSEwDXot8fsrM6t93NnAbcBXwaeA6MxtOdsR30xYhoWkMxcEXPe+REhMnEseBgej3uPt47dcn\ngdvc/S13fx3YRfW+i1Rpi5AQRUOxae+W0mwULe+4BEaAK4AfmdnHgQORY0PAD83swtp1LQbubHWF\ng4MDrb5kUpVKhZ89sB+AL13xka6uK6mZ0hLiTBDmXKHMtGxwEdOnT2XzyO1896kdrFuyluFZlvdY\nXYkTiYeAT5rZSO3zlWZ2A3DY3R82sx8ATwJ/BO5y90OtrnB0tPP7Ng89f5Snj7zCgg+9l5lT+7u6\nrrrBwYFEridJIc4EYc4V2kyzz5jDTZesYfO+77Fxz5ZgHvXoNKQtI+HuFeAfGi5+NnJ8M7C5o1tv\nk+6LkKJYeO780jw8WqgnU+m+CCmSsjzqUZhIaIuQIipDKAoTCW0RUlRFD0UhIqEtQoquyKEoRCS0\nRUgZFDUUwUdCW4SUSRFDEXwktEVI2RQtFEFHQluElFWRQhF0JLRFSJkVJRTBRkJbhPSCIoQi2Eho\ni5BeEXoogoyEtgjpNSGHIshIaIuQXhRqKIKLhLYI6WUhhiK4SGiLkF4XWiiCioS2CJGqkEIRVCS0\nRYi8I5RQBBMJbREi7xZCKIKJhLYIkYnlHYogIqEtQqS5PEMRRCS0RYi0llcoco+EtgiR+PIIRe6R\n0BYh0p6sQ5FrJLRFiHQmy1DkGgltESKdyyoUuUVCW4RI97IIRW6R0BYhkoy0Q5FLJLRFiCQrzVDk\nEgltESLJSysULSNhZn1m9l0z+28z22Vmf9lwfLWZ/bJ2/LNxblRbhEg60ghFnE3i88BZ7n4xsB64\npX7AzGYB/wj8NfBpYKOZndHsysZOjWuLEElR0qGIE4nFwCMA7v4kcFHk2CJgn7ufcvfjwGFgQbMr\ne/3NMUBbhEiakgxFnEhMA16LfH7KzKZMcux1YHqzKxs79ba2CJEMNIaiU/0xvuY4MBD5fIq7j0eO\nRf+0DwDHml3ZmX1jvOeNJ7nnnl+2NWiapkzpY3y8kvcYpwlxJghzLs3U3IK3z+P1sdc7/v44kRgB\nrgB+ZGYfBw5Ejv0C+DczOxOYCpwPHGx2Zd/85r/2dTiriOSgr1JpXjsz6wO28s59DSuBzwKH3f1h\nM/sK8FWgD/iWu/9nivOKSMZaRkJEelvuLxUXkbApEiLSlCIhIk0pEiLSVJyHQDsSeVTkAuAtYJW7\nH4kcXw2sAcaoPiqyM61Z2pjpVuBi4ETtos+5+4l3XVE6s30M2OTuyxouXwFsoHqedrj79izmaTHT\nDcBXgJdqF33V3Q+nPEs/8H1gDnAm1d8zP40cz/w8xZgp8/NUu90pwDbAgHHgWnd/JnK8rXOVWiSI\nvOaj9pvtltpl0dd8LATeA+wzs8fcfSzFeZrOVLMQ+JS7v5ryHKcxs68BV1N9xmr08n6qM34UeBMY\nMbP/cveX3n0t2cxUsxC42t1/lfYcEVcBL7v7NWb258CvgJ9Crudp0plq8jhPACuAirsvNrO/Ab7N\nO3/22j5Xaf5zI9HXfKQ9U23L+DBwu5ntM7OVGcxT9xzwhQkun0f1+SjHawHdByzJeSao/gZbb2Z7\nzWxdRvM8QPVvP6g+Jyf6F0pe56nZTJDPecLdf0J1S4fqlnM0crjtc5VmJBJ9zUcGM50N3Eb1b4dP\nA9eZ2XAGM+HuDwGnJjjUOO8JsjlPzWYCuA+4FlgGLDazyzOY56S7v2FmA8CDwNcjh3M5Ty1mghzO\nU2S2cTO7E7gVuDdyqO1zlWYkEn3NRwYznQRuc/e33P11YBfV+y7ylNd5auVWd3/V3U8BO4ELs7hR\nM5tN9f/LXe5+f+RQbuepyUyQ03mqc/cvA0PAdjObWru47XOV5n0Sib7mI4OZhoAfmtmFVM/LYuDO\nDGaKanxdyyFgrpnNoBqxpcDNec5kZtOAg2Z2PtV/0y4H7kh7iNr9WI8Ca919d8PhXM5Ts5nyOk+1\n274K+IC7b6J6B/3btQ/o4FylGYmHgE+a2Ujt85W1e3vrr/m4jeq/h/qAf3b3P6Y4S9yZfgA8CfyR\n6t8MhzKYKaoCYGZXAme7+3YzuxF4jOp52u7uLwQw03rgcaq/AX/u7o9kMMd6YAawwcy+UZtrG/me\np1Yz5XGeAH4M7DCzJ6j+Gb8e+KKZdXSu9NoNEWlKT6YSkaYUCRFpSpEQkaYUCRFpSpEQkaYUCRFp\nSpEQkaYUCRFp6v8A9Mn+VQ3Rv70AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x108c19d90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(-10, 10)\n",
    "y = 2*x\n",
    "y1 = -x + 3\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(x, y)\n",
    "plt.plot(x, y1)\n",
    "plt.xlim(0, 3)\n",
    "plt.ylim(0, 3)\n",
    "# draw axes\n",
    "plt.axvline(x=0, color='grey')\n",
    "plt.axhline(y=0, color='grey')\n",
    "plt.show()\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the solution (corresponding to the line crossing) is when $x=1$ and $y=2$. It confirms what we found with the matrix inversion!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BONUS: Coding tip - Draw an equation\n",
    "\n",
    "To draw the equation with Matplotlib, we first need to create a vector with all the $x$ values. Actually, since this is a line, only two points would have been sufficient. But with more complex functions, the length of the vector $x$ corresponds to the sampling rate. So here we used the Numpy function `arrange()` (see the [doc](https://docs.scipy.org/doc/numpy/reference/generated/numpy.arange.html)) to create a vector from $-10$ to $10$ (not included)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-10,  -9,  -8,  -7,  -6,  -5,  -4,  -3,  -2,  -1,   0,   1,   2,\n",
       "         3,   4,   5,   6,   7,   8,   9])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.arange(-10, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first argument is the starting point and the second the ending point. You can add a third argument to specify the step:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-10,  -8,  -6,  -4,  -2,   0,   2,   4,   6,   8])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.arange(-10, 10, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we create a second vector $y$ that depends on the $x$ vector. Numpy will take each value of $x$ and apply the equation formula to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-19, -17, -15, -13, -11,  -9,  -7,  -5,  -3,  -1,   1,   3,   5,\n",
       "         7,   9,  11,  13,  15,  17,  19])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.arange(-10, 10)\n",
    "y = 2*x + 1\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, you just need to plot these vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Singular matrices\n",
    "\n",
    "Some matrices are not invertible. They are called **singular**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "This introduces different cases according to the linear system because $\\bs{A}^{-1}$ exists only if the equation $\\bs{Ax}=\\bs{b}$ has one and only one solution. [The next chapter](https://hadrienj.github.io/posts/Deep-Learning-Book-Series-2.4-Linear-Dependence-and-Span/) is almost all about systems of linear equations and number of solutions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span class='notes'>\n",
    "    Feel free to drop me an email or a comment. The syllabus of this series can be found [in the introduction post](https://hadrienj.github.io/posts/Deep-Learning-Book-Series-Introduction/). All the notebooks can be found on [Github](https://github.com/hadrienj/deepLearningBook-Notes).\n",
    "</span>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
